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(54) IMAGE SYNTHESIS METHOD AND IMAGE SYNTHESIZER 
(57)Abstract: 

PROBLEM TO BE SOLVED: To obtain the image synthesis 
method and the image synthesizer in which plural images 
having different image states due to difference from an 
image pickup condition or the like are synthesized in an 
excellent way. 

SOLUTION: Gradation conversion sections 31, 32 correct 
input images a, b depending on image states in an 
overlapped image area. In this case, gradation conversion is 
applied to the input images a, b, to correct them so that the 
lightness in the overlapped image area of the two images is 
made equal. Furthermore, an image conversion synthesis 
section 50 conducts synthesis processing so that the center 
of the overlapped part comes to a joint and the input images 
a, b whose lightness is corrected by the gradation 

conversion sections 31 , 32 are converted by a conversion parameter and the result is 
synthesized into one image. 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may 

not reflect the original precisely. 

2 **** shows the word which can not be translated. 

3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] It is the image synthesizer unit characterized by performing it in 
an image synthesizer unit which compounds two or more images which have 
an image field which overlapped mutually, and creates one synthetic image 
with a large field angle after said image composition amends said two or 
more images according to an image condition in an image field where said 
two or more images overlapped. 

[Claim 2] It is the image synthesizer \mit characterized by performing it in 
an image synthesizer unit which compounds two or more images which have 
an image field which overlapped mutually, and creates one synthetic image 
with a large field angle after said image composition amends said two or 
more images according to an image condition of a core in an image field 
where said two or more images overlapped. 

[Claim 3] An image synthesizer unit according to claim 1 or 2 characterized 
by having a detection means to detect gradation of two or more of said 
images as said image condition, and an amendment means to amend said 
two or more images so that gradation of an image field which overlapped 
based on this detection result may become equal. 

[Claim 4] It is the image composition method characterized by performing it 
in an image composition method which compounds two or more images 
which have an image field which overlapped mutually, and creates one 
synthetic image with a large field angle after said image composition amends 
said two or more images according to an image condition in an image field 
where said two or more images overlapped. 

[Claim 5] It is the image composition method characterized by to perform it 
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in an image composition method which compounds two or more images 
which have an image field which overlapped mutually, and creates one 
synthetic image with a large field angle after said image composition amends 
said two or more images according to an image condition of a core in an 
image field where said two or more images overlapped. 
[Claim 6] An image composition method according to claim 4 or 5 
characterized by amending said two or more images so that gradation of an 
image field which detected gradation in said duplicate image field as said 
image condition, and overlapped based on this detection result may become 
equal. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[The technical field to which invention belongs] Especially this invention 
relates two or more images with which some images overlap about the image 
composition method and an image synthesizer unit to the image composition 
method and image synthesizer unit which compound a panorama image with 
a large field angle in piles. 
[0002] 

[Description of the Prior Art] What performs geometric conversion of affine 
transformation etc. and connects two images on a plane in the above image 
composition methods and an image synthesizer unit conventionally so that 
the point same about in the field where an image overlaps may be in 
agreement is known. 

[0003] This creates a synthetic image as performed geometric conversion of 
affine transformation etc. in the field where this image overlaps two input 
images which picturized the photographic subject as shown in drawing 23 
like drawing 24 and drawing 25 within the Umit of fi*ames fl and £2 with the 
electronic "still" camera etc., respectively and shown in drawing 26 . 
[0004] In addition, in this case, since the whole is dark, the place where the 
hght exposure at the time of an image pick-up is dark is amended so that it 
may become bright, and, as for the photographic subject of a fi-ame fl, an 
image like drawing 24 is picturized. Moreover, since the whole is bright, the 
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place where the light exposure at the time of an image pick-up is bright is 
amended so that it may become dark, and, as for the photographic subject of 
a frame £2, an image like drawing: 25 is picturized. 
[0005] 

[Problem(s) to be Solved by the Invention] However, there were the following 
troubles in the above-mentioned conventional example. 

[0006] Since the image pick-up conditions of each image differed by the factor 
of a photographic subject etc., even when the image concerning composition 
is connected with a sufficient precision, a joint will arise on a synthetic image. 
[0007] Even when the above-mentioned input image is connected notably 
[ when exposure conditions differ ] with a sufficient precision by especially 
this, it will become a synthetic image as shown in drawing 26 , and a joint 
will be conspicuous on a synthetic image with the difference in the 
brightness of the input image in the same photographic subject. 
[0008] Furthermore, when the property of an image surface illuminance 
ratio compounds two or more images picturized by the image pick-up optical 
system which has a difference remarkably on the outskirts of near an optical 
axis, a joint will arise by the difference in an image condition. 
[0009] Then, it is made in order that this invention may solve such a problem, 
and it aims at offering the image composition method which can compound 
two or more images which had a different image condition by the difference 
among image pick-up conditions etc. good, and an image synthesizer unit. 
[0010] 

[Means for Solving the Problem] In order to solve the above-mentioned 
technical problem, this invention has the following means. 
[00 ll] An image synthesizer unit according to claim 1 compounds two or 
more images which have an image field which overlapped mutually, and it is 
characterized by performing it, after said image composition amends said 
two or more images according to an image condition in an image field where 
said two or more images overlapped in an image synthesizer unit which 
creates one synthetic image with a large field angle. 

[0012] An image synthesizer unit according to claim 2 compounds two or 
more images which have an image field which overlapped mutually, and it is 
characterized by performing it, after said image composition amends said 
two or more images according to an image condition of a core in an image 
field where said two or more images overlapped in an image synthesizer unit 
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which creates one synthetic image with a large field angle. 
[0013] An image synthesizer unit according to claim 3 is characterized by 
having a detection means to detect gradation of two or more images as an 
image condition, and an amendment means to amend said two or more 
images so that gradation of an image field which overlapped based on this 
detection resiilt may become equal in an image synthesizer unit according to 
claim 1 or 2. 

[0014] An image composition method according to claim 4 compounds two or 
more images which have an image field which overlapped mutually, and it is 
characterized by performing it, after said image composition amends said 
two or more images according to an image condition in an image field where 
said two or more images overlapped in an image composition method which 
creates one synthetic image with a large field angle. 

[0015] An image composition method according to claim 5 compounds two or 
more images which have an image field which overlapped mutually, and it is 
characterized by performing it, after said image composition amends said 
two or more images according to an image condition of a core in an image 
field where said two or more images overlapped in an image composition 
method which creates one synthetic image with a large field angle. 
[0016] In an image composition method according to claim 4 or 5, an image 
composition method according to claim 6 detects gradation in said duplicate 
image field as said image condition, and is characterized by amending said 
two or more images so that gradation of an image field which overlapped 
based on this detection result may become equal. 
[0017] 

[Embodiment of the Invention] Hereafter, the gestalt of operation of this 
invention is explained to details with reference to a drawing. 
[0018] (Gestalt of the 1st operation) The gestalt of the 1st operation by this 
invention is explained first. 

[0019] Drawing 1 is configuration block drawing showing one gestalt of 
operation of the image synthesizer unit of this invention. 
[0020] This equipment compounds the input images a and b picturized with 
the electronic "still" camera, the video camera, etc., as a synthetic image c, it 
outputs to CRT, a printer, etc. and has a non-illustrated control section, and 
possesses the corresponding-points extract section 20 further controlled by 
this control section respectively, the parameter estimation section 40, the 
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gradation correction factor decision section 30, the gradation transducers 31 
and 32, and the image transformation composition section 50, and is 
constituted. 

[0021] the image field (henceforth the "dupHcation section") where some 
images overlapped the photographic subject as show the input images a and 
b to drawing 2 by be overlap and picturize within the Hmit of Frames fa and 
fb with the electronic "still" camera, the video camera, etc. here, respectively 
- an owner - it be a thing the bottom and it have become the image with 
which the exposure conditions at the time of an image pick-up differ as the 
gestalt of this operation be further showed in drawing 3 and drawing 4 . In 
addition, the case where the image data of the input images a and b is shade 
image data of N gradation, respectively is explained into the gestalt of the 
following operations. 

[0022] The corresponding-points extract section 20 judges reliability over 
this corresponding-points extract, and stores the coordinate in the input 
images a and b of corresponding points in internal memory while it extracts 
the corresponding points overlapped in the image between the input images 
a and b. Moreover, in case corresponding points are extracted, since the 
duplication section is not known beforehand, a predetermined field is set up 
as a template logging field into the input image a. 

[0023] moreover, when the input images a and b are what is arranged in 
order of the left and the right, the corresponding-points extract section 20 
this template logging field 90 percent is made into the template logging field 
(the field T of drawing 5 ) in the input image a firom ten percent of a 
perpendicular direction 30 percent from the horizontal right end of the input 
image a. As the slash of drawing 6 shows, the small field Ta of the magnitude 
of about ten percent of image size is started as a template per block. In 
addition, in being what is arranged in order of the upper bottom at this time 
a and b, for example, input images, let 90 percent be a template logging field 
in the input image a from 30 percent fi:om the horizontal lower limit of the 
input image a, and ten horizontal percent. 

[0024] Furthermore, since the duplication section does not understand the 
corresponding-points extract section 20 beforehand in the input image b, it 
sets up the predetermined field searched for the point corresponding to said 
started template as a seek area. 

[0025] This seek area is set up like the field S shown by drawing 7 , and is set 
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in the gestalt of this operation. To a left location 50 percent from the location 
in the input image a of the horizontal left end of the input image b to a 
template It is perpendicularly set as the field from the location in the input 
image a of a template to the location of **ten percent, and the dupUcation 
section of the input images a and b is horizontal, this field setup is 50 or less 
percent, and, perpendicularly, it is based on the conditions of not shifting, 
**ten percent or more. In addition, what is necessary is just to change a 
setup of the seek area of this corresponding-points extract, when the 
duphcation conditions the input images a and b are assumed to be differ. 
[0026] The parameter estimation section 40 presumes the parameter which 
performs image transformation from a corresponding-points vector, and 
presumes the parameter of coordinate transformation from the 
corresponding-points location extracted in the corresponding-points extract 
section 20. 

[0027] Moreover, supposing the parameter estimation section 40 performs 
this coordinate transformation by affine transformation and the input image 
b is in one m times the relation [ theta rotation a parallel displacement (dx, 
dy), and ] of this that carried out expansion conversion to the input image a, 
the point in the input image a (xa, ya) is equivalent to the point in the input 
image b of following formula ** (xb, yb). 
[0028] 

xb=(cos theta-xa+sin theta-ya-dx) xm = A-xa+B-ya+C yb=(-sin theta-xa+cos 
thetaya-dy) xm = -B-xa+A-ya+D .... They are formula ** however 
A=m-costheta, B=sintheta, c=-m-dx, and D=-m-dy. 

[0029] The parameter estimation section 40 presumes the parameters A, B, 
C, and D at this time with a least square method. However, when the 
number of the corresponding points which at least two pairs of coordinates of 
corresponding points are required, and were able to be found in quest of the 
parameter is one, and making into a parallel displacement parameter the 
average value of the corresponding-points vector which was able to be found 
to 1/2 image, namely, setting the average value of a corresponding-points 
vector to (ax, ay), a parameter is set up and outputted like formtila **. 
[0030] 

A= 1, B= 0, C=-ax, D=-ay .... In case the parameter which outputted the 
message to CRT, ended processing and was presumed predicts the 
duplication section, it is used without formula **, in addition the parameter 
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estimation section 40 performing future processings, when one pair cannot 
be found by corresponding points, either. 

[0031] After the image transformation composition section 50 changes the 
input images a and b which synthetic processing was performed [ images ] 
and had brightness amended by the gradation transducers 31 and 32 with a 
conversion parameter so that the center of the dupUcation section may 
become a joint, and it sets up the image field of the synthetic image c, it is 
compounded in one image. 

[0032] Moreover, the image transformation composition section 50 sets up 
this image field on the basis of the system of coordinates of the input image a, 
and sets it up like the field shown with the dashed line of drawing 8 . That is, 
make a left end into the left end coordinate of the input image a, and let a 
right end be the coordinate value of the larger one among the coordinates 
which changed and asked the coordinate of the input image a for the pixel of 
the top right corner of the input image b, and a lower right edge. 
[0033] Furthermore, the image transformation composition section 50 
performs conversion on the coordinate of the input image a of the input 
image b from a coordinate using the inverse transformation of the affine 
transformation of said formula **. That is, if the parameter of inverse 
transformation is made into A\ B\ C\ and D', formula ** shown below will be 
changed. 
[0034] 

xa=A', xb+B', andyb+C ya=-B', xb+A', and yb+D' .... Formula ** however 
A'=A/(A2+B-2), B'=-B/(A2+B-2), C'=(-AC+BD)/(A2+B-2), 

D'=(-BC+AD)/(A2+B-2) 
It comes out. 

[0035] In addition, let the coordinate value of the smaller one, and a lower 
limit be the coordinate values of the larger one among the coordinates which 
changed and asked the coordinate of the input image a for the pixel of the 
lower limit coordinate value of the input image a and an input image b lower 
right edge, and a lower-left edge among the coordinates which the upper 
limit of the synthetic image c changed the pixel of the upper limit coordinate 
value of the input image a and the top right corner of the input image b, and 
an upper-left edge into the coordinate of the input 

[0036] Here, the image of the synthetic image c of the input images a and b 
called for by the image transformation composition section 50 as mentioned 
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above becomes like drawing 9 . in addition, the portion shown with the slash 
in drawing 9 - the input images a and b - allocation of a pixel should also do 
either - dummy pixels (for example, white pixel etc.) enter in the field which 
is not. 

[0037] The gradation transducers 31 and 32 are what amends the image of 
the input images a and b according to the image condition in the duplicate 
image field. It has a function as an amendment means to amend said two or 
more images so that the gradation of the image field which overlapped based 
on the detection resvdt of the gradation correction factor decision section 30 
as a detection means to mention later may become equal. In the gestalt of 
this operation, gray scale conversion is performed to the input images a and 
b, respectively, and it amends so that brightness in the image field in which 
two images overlap may be made equal. 

[0038] In addition, if the gradation average of the pixel value judged as the 
gradation transducers 31 and 32 being in duplication circles is set to Pa and 
Pb, based on the following formula, it will amend in quest of pixel value pa' 
and pb*. 
[0039] 

pa'=paxN/(N+Pa-Pb) pb*=(pb+Pa-Pb) xN/(N+Pa-Pb) 

.... For formula **, however pa and pb, the pixel value of the input images a 
and b and N are the number of gradation, respectively. 

[0040] In addition, although Pa and Pb should become equal if the input 
images a and b are picturized on the same exposure conditions here, since 
the photographic subject in the duplication section is in agreement As 
drawing 3 and drawing 4 showed, when it is picturized When the brightness 
of a photographic subject is taken along a horizontal axis and the gradation 
of the image data at the time of an image pick-up is taken along an axis of 
ordinate, the property at the time of the image pick-up of the input images a 
and b is shown in drawing 10 (however, in drawing 10 , the property over the 
input image b is expressed with 10b for the property over the input image a 
to 10a). 

[0041] So, with the gestalt of this operation, by the gradation transducers 31 
and 32, this property is approximated like drawing 11 and considered, and 
gradation amendment is performed according to formula ** so that the 
gradation of the photographic subject in the duplication section between each 
input image may become equal (however, in drawing 11 , the property over 
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the input image b is expressed with lib for the property over the input 
image a to 11a). That is, since only the part of an arrow head increases only 
as for the part of the dynamic range [ raising and ] of a pixel value of the 
arrow head of drawing 11 , the gradation of the input image b is normalized 
to the gradation of a basis. 

[0042] The gradation correction factor decision section 30 judges whether the 
coordinate value in each input image is in the duplication section using the 
output parameter of the parameter estimation section 40. That is, according 
to formula **, affine transformation is performed, each coordinate value of 
the input image a is changed into the coordinate value of the input image b, 
and this coordinate value judges whether it is the inside of the field of the 
input image b. 

[0043] Moreover, the gradation correction factor decision section 30 has a 
function as a detection means to detect the gradation of the input images a 
and b. Although the gradation averages Pa and Pb of each pixel value are 
calculated, respectively after it adds the pixel value judged that is in 
duplication circles about the input images a and b, respectively and 
processing finishes about all the pixels of the duplication section In the case 
of the image with which a difference has the property of an image surface 
illuminance ratio remarkably on the outskirts of near an optical axis, the 
average is calculated near the center of the duplication section shown in the 
field V of drawing 12 without calculating the average in aU the fields of the 
duplication section. 

[0044] Furthermore, the gradation correction factor decision section 30 

determines the correction factor at the time of performing gradation 

amendment in the gradation transducers 31 and 32 from the image data of 

the portion which the input images a and b overlap. However, this correction 

factor is expressed with coefficient (Pa-Pb) and N/(N+Pa-Pb) in said formula 
** 

[0045] in addition , it be convenient also from the point that an image surface 
illuminance ratio become [ as opposed to / in near the joint which be the 
center of the duplication section depending on the distance from an optical 
axis to the image point like / in a lens with the image surface illuminance 
ratio common near the optical axis of image pick-up optical system , and in 
the circumference / drawing 13 although synthetic processing be perform so 
that the center of the duplication section may become a joint by the image 
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transformation composition section 50 as mentioned above with the gestalt 
of this operation / the same photographic subject ] equal . 
[0046] That is, since it is hQ<hQ' when the center O of the input images a 
and b and distance from O' are made into hQ and hQ* at Point Q in drawing 
14 , respectively, the brightness of a photographic subject is not in agreement, 
but since it is hR=hR' when the center O of the input images a and b and 
distance from O' are made into hR and hR' at the point R near the center of 
the duplication section, respectively, the brightness of a photographic subject 
is in agreement. 

[0047] therefore, in case synthetic processing of the input image picturized 
by image pick-up optical system, for example like a wide angle lens in which 
the property of an image surface illuminance ratio has a difference 
remarkably on the outskirts of near an optical axis as shown in drawing 13 is 
carried out The average of each input image is computed in all the fields of 
the duplication section, it can amend with a precision sufficient [ direction ] 
computing the average in the portion whose brightness of the photographic 
subject near the center of the duplication section corresponded, and 
performing gradation amendment, and the joint produced according to 
exposure conditions stops being conspicuous rather than it performs 
gradation amendment. 

[0048] Next, actuation of the image sjnathesizer unit of the gestalt of 
operation of the 1st of this invention when generating the synthetic image c 
from the input images a and b is explained. 

[0049] First, the actuation at the time of performing a corresponding-points 
extract in the corresponding-points extract section 20 is explained. 
[0050] Drawing 15 is a flow chart which shows the processing algorithm of 
the corresponding-points extract section 20. 

[0051] In this drawing, the corresponding-points extract section 20 sets up a 
template logging field from the input image a first (step S2l). 
[0052] And processing by the following step step S22 - step S24 is performed 
about all the started templates. 

[0053] First, the seek area which searches for the point corresponding to the 
started template is set up from the input image b (step S22). Let the field S 
shown with the slash which makes a predetermined field a seek area since a 
duplication field is not known beforehand, for example, is shown in drawing 
7 be a seek area at this time. 
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[0054] Next, the template is shifted in parallel within this seek area, and the 
difference of the input images a and h is calculated. And total of the absolute 
value of a difference makes the location used as min a corresponding-points 
location (step S23). 

[0055] And reliability over the result of step S23 is judged (step S24). 
[0056] The judgment of this rehabihty is performed using total of the 
absolute value of total of the absolute value of the difference used as the 
minimum value, and a difference small to the 2nd. When there is a difference 
of a value with total of the minimum value of total of the absolute value of a 
difference of the absolute value of below the 1st predetermined threshold and 
a difference small to the 2nd and the minimum value beyond the 2nd 
predetermined threshold, in the case of the gestalt of this operation, it judges 
that reliability is in corresponding points, and it stores the coordinate in the 
input images a and b of corresponding points in the memory in the 
corresponding-points extract section 20. 

[0057] In addition, although total of the absolute value of a difference made 
the location used as min the corresponding-points location by the above here, 
it is good also considering the location where a correlation operation is 
performed, for example and a correlation value serves as max as a 
corresponding-points location. 

[0058] Moreover, although the above-mentioned processing extracted the 
corresponding points of the input images a and b from image data, two 
images may be displayed on a display, and the same point in an image may 
be specified and extracted with cursor etc., for example. 

[0059] Next, the actuation in the gradation correction factor decision section 
30 and the gradation transducers 31 and 32 is explained. 
[0060] Drawing 16 is a flow chart which shows the processing algorithm of 
the gradation correction factor decision section 30 and the gradation 
transducers 31 and 32. 

[0061] In this drawing, it first judges whether the gradation correction factor 
decision section 30 has a coordinate value in each input image in the 
duplication section using the output parameter of the parameter estimation 
section 40 (step Si). That is, according to formula **, affine transformation is 
performed, each coordinate value of the input image a is changed into the 
coordinate value of the input image b, and this coordinate value judges 
whether it is the inside of the field of the input image b. 
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[0062] Next, after it adds the pixel value judged that is in duplication circles 
about the input images a and b, respectively and processing finishes about 
all the pixels of the duplication section, the gradation averages Pa and Pb of 
each pixel value are calculated, respectively (step S2). 

[0063] And if the gradation correction factor decision section 30 calculates 
coefficient (Pa-Pb) and N/(N+Pa-Pb) in the further above-mentioned formula 
** (step S3), the gradation transducers 31 and 32 will process gradation 
amendment based on formula ** by the full screen about each input image 
(step S4). 

[0064] In addition, although the average of the pixel value of the duplication 
section of the input images a and b was used for gradation amendment in the 
gestalt of the above-mentioned implementation, respectively, even if it uses 
the median of the pixel value of the duplication section, for example, an 
almost equivsdent effect is acquired. 

[0065] Moreover, although the judgment of a duplication section pixel which 
performs a pixel average is performed using the output parameter of the 
parameter estimation section 40, you may judge directly fi:om the 
corresponding-points coordinate which is the output of the 
corresponding-points extract section 20. What is necessary is to add 
separately each pixel value of the corresponding points which were able to 
perform the corresponding-points extract, and just to calculate each average 
at this time. 

[0066] Actuation concerning generation of the synthetic image c by the image 
transformation composition section 50 is explained to the last. 
[0067] Drawing 17 is a flow chart which shows the processing algorithm of 
the image transformation composition section 50. 

[0068] In this drawing, the image transformation composition section 50 sets 
up the image field of the synthetic image c first (step S5l). 
[0069] Next, as it indicated that the location of a joint took the lead in the 
duplication section with the dashed line L of drawing 8 , it sets up (step S52). 
That is, let the average with the coordinate value of the smaller one be the 
location of a joint among the coordinates which changed and asked the 
coordinate of the input image a for the right end coordinate value of the 
input image a, and the pixel of the upper left edge of the input image b, and a 
lower left edge. 

[0070] And a pixel value is calculated from the field of the synthetic image c 
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set up at step S51, respectively (step S53). 

[0071] First, to that which is in the field of the input image a, the pixel value 
of the input image a is written in as it is (step S54). 

[0072] And a coordinate value is changed into the coordinate value of the 
input image b, and the pixel value of the input image b of the location is 
written in as it is (step S55). When the pixel value of the input image a is 
already written in at this time, it is made to write in only about the thing on 
the right of the location of a joint. 

[0073] Thus, the image of the synthetic image c for which it asked becomes 
like drawing 9 . the portion shown with the slash by drawing 9 - the input 
images a and b allocation of a pixel should also do either - dummy pixels 
(for example, white pixel etc.) enter in the field which is not. 
[0074] In addition, in the above, when the image surface illuminance ratio of 
image pick-up optical system is known beforehand, it cannot be 
overemphasized that you may amend so that the image surface illuminance 
ratio of each input image may become 1 in all image fields as pretreatment 
which performs image composition processing of this invention. 
[0075] Moreover, although the image at the time of composition assigned one 
of the input images a and b above in the image transformation composition 
section 50 of the gestalt of this operation, the average of the input images a 
and b may be assigned in the duplication department. Moreover, weight may 
be horizontally attached to adjustable to the pixel value of the input images a 
and b in the duplication department, and the average may be assigned. 
[0076] Since gradation amendment is performed in the field in which the 
same photographic subject of the duplication section of an input image is 
picturized according to the gestalt of this operation as explained above, even 
when the exposure conditions of an input image differ, an image can be 
compounded so that a joint may not be conspicuous. 

[0077] And since it is amending using the image data near the center of the 
image-overlap section, also when the property of an image surface 
illuminance ratio picturizes by the image pick-up optical system which has a 
difference remarkably on the outskirts of near an optical axis, an image can 
be compounded so that a joint may not be conspicuous. 

[0078] (Gestalt of the 2nd operation) Next, the gestalt of operation of the 2nd 
of this invention is explained. 

[0079] Since it is that of this appearance mostly and only the method of 
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gradation amendment differs from the configuration of the gestalt of the 1st 
operation shown in drawing 1 , the configuration of the 2nd of the image 
synthesizer unit of the gestalt of operation of this invention explains the 
method of different gradation amendment from the gestalt of the 1st 
operation hereafter. 

[0080] Although gradation amendment was performed in the gradation 
transducers 31 and 32 with the gestalt of said 1st operation based on formula 
[ average / of the pixel value judged that is in dupUcation circles / gradation / 
using Pa and Pb ] ** the gestalt of the 2nd operation " the input images a 
and b - although it is dark, based on formula ** of a degree, gradation 
amendment is performed for the gradation average in quest of pixel value pa' 
and pb' as Pal, Pbl, Pa2, and Pb2 with the comparatively bright thing of a 
pixel value about each. 
[0081] 

pa'=paxK2 .... Formula however Kl=(Pa2-Pal)/(Pb2-Pbl), K0= 

(PalxPb2-Pa2xPbl/(Pb2-Pbl), K2=N/(Klx(N-Pbl)+Pal)) 
pb'=(Klxpb+K0)+K2 
It comes out. 

[0082] Hereafter, explanation about derivation of this formula ** is given. 
[0083] First, since the photographic subject in the duplication section 
concerning image composition is in agreement, if the input images a and b 
are picturized on the same exposure conditions, the gradation average Pa 1, 
and Pbl, Pa2 and Pb2 should become equal. However, when it is picturized 
like the gestalt of the 1st operation, and the brightness of a photographic 
subject is taken along a horizontal axis and the gradation of the image data 
at the time of an image pick-up is taken along an axis of ordinate, the 
property at the time of the image pick-up of the input images a and b is 
shown [ as drawing 3 and drawing 4 explained, ] in drawing 10 . 
[0084] With the gestalt of this operation, the brightness of the photographic 
subject at the time of an image pick-up as shown in drawing 10 , and the 
property of the gradation of image data are approximated like drawing 18 , 
and are considered, and gradation amendment is performed so that the 
gradation of the photographic subject in the duplication section between each 
input image may become equal (however, in drawing 18 , the property over 
the input image b is expressed with 18b for the property over the input 
image a to 18a). That is, when the photographic subject equivalent to the two 
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gradation averages sets brightness to LI and L2 of drawing 18 , only the part 
of the arrow heads Dl and D2 of drawing 18 raises the gradation of the input 
image b, respectively. And the gradation of the input image b is amended 
Hke following formula **. 
[0085] 

pb'=Klxpb+KO .... Formula ** however Kl=(Pa2 Pal)/(Pb2-Pbl), K0= 

(PalxPb2-Pa2xPbl/(Pb2-Pbl)) 

It comes out. 

[0086] Furthermore, if only the part whose dynamic range of a pixel value 
increased in this formula ** is normahzed to the gradation of a basis, said 
formula ** will be drawn. 

[0087] In addition, the gradation correction factor decision section 30 
determines the correction factor at the time of performing gradation 
amendment in the gradation transducers 31 and 32 like the gestalt of the 1st 
operation from the image data of the duplication section of said input images 
a and b. 

[0088] However, the correction factor in the gestalt of the 2nd operation here 
is expressed with the coefficients KO, Kl, and K2 in said formula **. 
[0089] Next, actuation concerning the gestalt of operation of the 2nd of this 
invention is explained focusing on the actuation in the gradation correction 
factor decision section 30 and the gradation transducers 31 and 32. 
[0090] Drawing 19 is a flow chart which shows the processing algorithm of 
the gradation correction factor decision section 30 and the gradation 
transducers 31 and 32. 

[0091] In this drawing, it first judges whether the gradation correction factor 
decision section 30 has a coordinate value in each input image in the 
duplication section using the output parameter of the parameter estimation 
section 40 (step Sll). That is, according to formula **, affine transformation 
is performed, each coordinate value of the input image a is changed into the 
coordinate value of the input image b, and this coordinate value judges 
whether it is the inside of the field of the input image b. 

[0092] next - although the pixel value judged that is in duplication circles is 
added about the input images a and b, respectively and it is as dark as the 
comparatively bright thing of a pixel value with the gestalt of this operation 
- the gradation averages Pal, Pbl, Pa2, and Pb2 - the input images a and b 
it calculates about each (step S12). 
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[0093] At this time, a pixel [ than a predetermined threshold ] value with the 
larger pixel value of the input image a and the pixel value of the input image 
b corresponding to it are added, respectively, and a pixel value smaller than 
a predetermined threshold and the pixel value of the input image b 
corresponding to it are added similarly, respectively. And after processing 
finishes about all the pixels of the duplication section, the gradation 
averages Pal, Pbl, Pa2, and Pb2 corresponding to two brightness of each 
pixel value are calculated, respectively. 

[0094] And coefficients KO, Kl, and K2 are beforehand calculated in the 
gradation correction factor decision section 30 (step SI 3), and gradation 
amendment is processed [ by the gradation transducers 31 and 32 ] based on 
formula ** by the full screen about each input image (step S14). 
[0095] (Gestalt of the 3rd operation) Next, the gestalt of operation of the 3rd 
of this invention is explained. 

[0096] The configuration of the 3rd of the image synthesizer unit of the 
gestalt of operation of this invention is shown in drawing 20 . Although it is 
the same as that of the 1st configuration of the gestalt of operation almost, in 
case [ at which the configuration of the gestalt of this operation was shown in 
drawing 1 ] it asks for a gradation correction factor in the gradation 
correction factor decision section 30, it uses the corresponding-points 
coordinate which is the output of the corresponding-points extract section 20 
as it is with the gestalt of this operation. 

[0097] Hereafter, since only the method of gradation amendment differs firom 
the gestalt of operation of the 1st of this invention, only the method of this 
gradation amendment is explained. 

[0098] With the gestalt of this operation, the property at the time of the 
image pick-up of the input images a and b shown in drawing 10 like the 
gestalt of said 2nd operation is approximated Hke drawing 18 , and is 
considered, and gradation amendment is performed so that the gradation of 
the photographic subject in duplication between each input image may 
become equal. Therefore, in quest of pixel value pa' and pb', gradation 
amendment is performed by the method shown in formula ** of the following 
like formula ** and formula **. 
[0099] 

pa'=paxK2' pb*=(Kl*xpb+KO0 xK2' .... Formula **, however 
K2'=N/(Kl'x(N-PbO +PaO 
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It comes out. 

[0100] However, it asks for coefficient KO* of formvda **, Kl', and K2' in the 
gestalt of this operation, using the corresponding-points coordinate which is 
the output of the corresponding-points extract section 20 in the gradation 
correction factor decision section 30 as it is. 

[OlOl] Hereafter, the method is explained. First, it asks for coefficient KO' of 
following formula ** corresponding to formtxla [ of formula ** ] **, and Kl'. 
[0102] 

pb'=Kl'xpb+KO' .... formula ** - here, since gradation is equal when it 
picturizes on the same exposure conditions in that the input images a and b 
correspond Coefficient KO' and Kl' are calculated from the pixel value of all 
the corresponding points extracted so that the absolute value of the square of 
the difference of the pixel value and the pixel value of the input image a 
calculated by having assigned it to pb of the right-hand side of formula ** 
among the pixel values of the corresponding-points coordinate which is the 
output of the corresponding-points extract section 20 might serve as min. 
[0103] And the average of the pixel value of all the corresponding points 
extracted by coincidence is calculated as Pa' and Pb'. 

[0104] The above processing is performed in the gradation correction factor 
decision section 30, and gradation amendment of each input image is 
performed according to formxda ** in the gradation transducers 31 and 32. 
[0105] (Gestalt of the 4th operation) Next, the gestalt of operation of the 4th 
of this invention is explained. 

[0106] The configuration of the 4th of the image synthesizer unit of the 
gestalt of operation of this invention is the same as the configuration of the 
1st of the gestalt of operation of this invention shown in drawing 1 , and 
since only the methods of gradation amendment differ, the method of this 
gradation amendment is explained. 

[0107] With the gestalt of this operation, although the gradation which the 
input images a and b can take is N gradation, the method of the gradation 
amendment for the case where the min of the image data in all image fields 
and maximum are Namin, Namax, Nbmin, and Nbmax, respectively is 
explained. 

[0108] Namely, the case where the property at the time of the image pick-up 
of the input images a and b shown in drawing 10 is drawing 21 is considered. 
Furthermore, this property is approximated like drawing 22 , and gradation 
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amendment is performed so that the gradation of the photographic subject in 
the duplication section between each input image may become equal 
(however, it sets to drawing 21 and drawing 22 ). The property over the input 
image b is expressed with 21b and 22b for the property over the input image 
a to 21a and 22a. 

[0109] Although the processing algorithm of the gradation correction factor 
decision section 30 at this time and the gradation transducers 31 and 32 is 
the same as that of what was shown in drawing 16 , before processing step 
SI, the min about all the image fields of the input images a and b and 
maximum are calculated first. And the same processing as the gestalt of 
operation of the 1st of this invention shown in drawing 16 is performed. 
However, in order to extend the dynamic lens of the parenchyma of a pixel 
value to N gradation, the min of image data, Maximums Namin, Namax, and 
Nbmin, and Nbmax are used for normalization. That is, following formula ** 
performs gradation amendment in quest of pixel value pa' and pb'. 
[0110] 

pa'=(pa-Nmin) xN/(Nmax-Nmin +l) pb*=(pb+Pa-Pb-Nmin) xN/(Nmax-Nmin 
+1) 

.... Formula **, however Nmin=MIN (Namin, Nbmin+Pa-Pb) 
Nmax=MAX (Namax, Nbmax+Pa-Pb) 

It comes out, and it is and, in MINO, the minimum value and MAXO express 
maximum. 

[0111] As explained above, even when there are few dynamic ranges of the 
image data of the input images a and b than the gradation which can be 
taken as image data according to the gestalt of this operation, using the 
gradation average value in the duplication section of the input images a and 
b, and the min of the image data in all the image fields of the input images a 
and b and maximum, gradation amendment is performed, and since it 
compounds, the synthetic image with contrast high moreover with which a 
joint is not conspicuous is obtained. 

[0112] In addition, this invention is not limited to the gestalt of the 
above-mentioned implementation, but various deformation implementation 
is possible for it. 

[0113] For example, although explained for the shade image in explanation 
in the gestalt of each above operation, it cannot be overemphasized that a 
color picture is also applicable. Although it arises also in a color-balance that 
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it is the same as that of the inequality of exposure conditions explained to 
the trouble of said conventional technology when a white balance is 
automatically performed, respectively at the time of the image pick-up of the 
input images a and b at that time For example, what is necessary is just to 
amend image data like the case of a shade image about each of the RGB 
component of a color from the image data of the pixel from which the image 
data of duplication circles of an image or corresponding points was extracted. 
[0114] 

[Effect of the Invention] According to this invention explained in full detail 
above, the following effects are done so by having made it the configuration 
mentioned above. 

[0115] Since according to invention according to claim 1 it carries out after 
amending an image according to the image condition in the image field 
where two or more images overlapped image composition, in case two or 
more images which had a different image condition by the difference among 
image pick-up conditions etc. are compounded, the image synthesizer unit in 
which image composition is possible good can be offered so that a joint may 
not be conspicuous. 

[0116] Since according to invention according to claim 2 it carries out after 
amending an image according to the image condition of the core in the image 
field where the image each overlapped image composition, when the property 
of an image surface illuminance ratio compounds two or more images 
picturized by the image pick-up optical system which has a difference 
remarkably on the outskirts of near an optical axis, for example, the image 
synthesizer unit which can perform image composition good so that a joint 
may not be conspicuous can be offered. 

[0117] Since according to invention according to claim 3 said two or more 
images are amended so that the gradation of the image field which detected 
the gradation of two or more images as an image condition, and overlapped 
based on this detection result may become equal, the image synthesizer unit 
which also in the case of the image picturized on the conditions from which 
exposure conditions differ, for example can perform image composition so 
that a joint may not be conspicuous can be offered. 

[0118] Since according to invention according to claim 4 it carries out after 
amending an image according to the image condition in the image field 
where two or more images overlapped image composition, also in case two or 
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more images which had the image condition which changes with differences 
among image pick-up conditions etc. are compounded, the image composition 
method compoundable good can be offered so that a joint may not be 
conspicuous. 

[0119] Since according to invention according to claim 5 it carries out after 
amending an image according to the image condition of the core in the image 
field where the image overlapped image composition, when the property of 
an image surface illuminance ratio compounds two or more images 
picturized by the image pick-up optical system which has a difference 
remarkably on the outskirts of near an optical axis, for example, the image 
composition method which can compound an image good so that a joint may 
not be conspicuous can be offered. 

[0120] Since according to invention according to claim 6 said two or more 
images are amended so that the gradation of the image field which detected 
the gradation of two or more images as an image condition, and overlapped 
based on this detection result may become equal, the image composition 
method which also in the case of the image picturized on the conditions fi-om 
which exposure conditions differ, for example can perform image composition 
so that a joint may not be conspicuous can be offered. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing l] It is the block diagram showing the configuration of the 1st of 
the gestalt of operation of this invention. 

[Drawing 2l It is drawing showing the image pick-up condition of the image 
to compound. 

[Drawing 3] It is drawing showing an example of an input image. 
[Drawing 4] It is drawing showing an example of an input image. 
[Drawing 5] It is drawing showing how to start the template in the case of a 
corresponding-points extract. 

[Drawing 6] It is drawing showing the template in the case of a 
corresponding-points extract. 

[Drawing 7] It is drawing showing the setting method of the seek area in the 
corresponding-points extract of this invention. 

[Drawing 8] It is drawing explaining the field of the synthetic image of this 
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invention. 

[Drawing 9] It is drawing showing an example of the synthetic image of this 
invention. 

[Drawing 10] It is drawing showing the brightness of the photographic 
subject at the time of an image pick-up, and the property of the gradation of 
image data. 

[Drawing 11] It is drawing explaining the method of gradation amendment 
of the gestalt of the 1st operation. 

[Drawing 12] It is explanatory drawing in the case of asking for a gradation 
average near the center of the duplication section. 
[Drawing 13] It is drawing showing the property of the image surface 
illuminance ratio of image pick-up optical system. 

[Drawing 14] It is drawing explaining the difference in the brightness by the 
property of the image surface illuminance ratio of an input image. 
[Drawing 15] It is the flow chart which shows the processing algorithm of the 
corresponding-points extract section. 

[Drawing 16] It is the flow chart which shows the processing algorithm of 
gradation amendment. 

[Drawing 17] It is the flow chart which shows the processing algorithm of the 
image transformation composition section. 

[Drawing 18] It is drawing explaining the method of gradation amendment 
of the gestalt of the 2nd operation. 

[Drawing 19] It is the flow chart which shows the processing algorithm of 
gradation amendment of the gestalt of the 2nd operation. 
[Drawing 20] It is the block diagram showing the 3rd configuration of the 
image synthesizer unit of the gestalt of operation. 

[Drawing 21] It is drawing showing the brightness of the photographic 
subject at the time of the image pick-up at the time of picturizing with little 
gradation, and the property of the gradation of image data. 
[Drawing 22] It is drawing explaining the method of gradation amendment 
of the gestalt of the 4th operation. 

[Drawing 23] It is drawing showing the image pick-up condition of the image 
which the former compounds. 

[Drawing 24] It is drawing showing an example of the conventional input 
image. 

[Drawing 25] It is drawing showing an example of the conventional input 
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image. 

[Drawing 26] It is drawing showing an example of the conventional synthetic 
image. 

[Description of Notations] 

20 Corresponding-Points Extract Section 

30 Gradation Correction Factor Decision Section 

31 32 Gradation transducer 

40 Parameter Estimation Section 

50 Image Transformation Composition Section 
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